Achieving Load Balancing of HDFS Clusters Using Markov Model

The combination of Hadoop and HDFS is becoming a defacto standard system in handling big data. HDFS is a distributed file system that is designed for big data. In HDFS, a file consists of multiple large sized blocks. A central management of HDFS tries to scatter these multiple blocks on different nodes to maximize the I/O throughput. Hadoop is a framework that supports data intensive parallel applications and runs on the top of HDFS. In Hadoop, a user-level job is usually split into small tasks and each task is assigned to cluster nodes where necessary data is placed. If the cluster nodes with necessary data is fully occupied by other jobs, the task is assigned to other clusters that might not hold necessary input data. In this case, these clusters need to transfer the input data from the other nodes at the cost of network traffic. Hadoop job scheduling is basically designed to reduce the network traffic. In other words, Hadoop prefers the nodes that holds the necessary input data when doing job scheduling. The ill-balanced placement of hot data would increase network traffic and affect the overall system performance as well. Therefore, well-balanced placement of hot data is critical to improve Hadoop performance. To achieve well-balance of hot data, the behaviors of system are monitored and some static rules are enforced by human beings in run time. The migration under these static rules tends to be conservative and progressed in slow pace in order to minimize the network traffic overhead. These characteristics of static migration make it unsuitable for the case of adding new cluster nodes, which requires rapid data migration and usually large volume of migration. In this study, we use Markov Model to achieve rapid migration of large data to new nodes.